What Do Thinking-Aloud Participants Say? A Comparison of Moderated and Unmoderated Usability Sessions

The value of thinking aloud in usability tests depends on the content of the users’ verbalizations. We investigated moderated and unmoderated users’ verbalizations during relaxed thinking aloud (i.e., verbalization at levels 1 through 3). Verbalizations of user experience were frequent and mostly relevant to the identification of usability issues. Explanations and redesign proposals were also mostly relevant, but infrequent. The relevance of verbalizations of user experience, explanations, and redesign proposals showed the value of relaxed thinking aloud but did not clarify the tradeoff between rich verbalizations and test reactivity. Action descriptions and system observations – two verbalization categories consistent with both relaxed and classic thinking aloud – were frequent but mainly of low relevance. Across all verbalizations, the positive or negative verbalizations were more often relevant than those without valence. Finally, moderated and unmoderated users made largely similar verbalizations, the main difference being a higher percentage of high-relevance verbalizations by unmoderated users.